Quasi-experiments in epidemiology

Lee Kennedy-Shaffer, PhD

2025-06-10

How Will I Know?

Image of Whitney Houston's How Will I Know album cover

About Me

  • Assistant Professor at Yale School of Public Health

  • Teach statistical modeling and study design

  • Research focus on infectious disease study design and cluster-randomized trials

Image of presenter—32-year-old white man with dark hair—holding a water pump

Rise of Quasi-Experiments

Citation for Nobel Memorial Prize in Economic Sciences from 2021

Text from the final paragraph of popular press release for Nobel Memorial Prize in Economic Sciences from 2021

QEs in Economics and Political Science

Title and abstract for Card and Krueger (1994)

Title and abstract for Abadie and Gardeazabal (2003)

Title for Abadie et al. (2010)

QEs in Epidemiology and Public Health

Title of Craig et al. (2017)

Title of Wing et al. (2018)

QEs in Epidemiology and Public Health

Title and citation of Waddington et al. (2017)

Citation for Matthay and Glymour (2022)

Considering the Role of Evidence

Title of Nianogo et al. (2023)

Title of Matthay et al. (2020)

  • Allows use of routinely-collected data

  • Evaluates interventions in-context

  • Provides “real world evidence”/population impact

  • Answers questions randomized trials and observational studies cannot

  • But … has threats to internal and external validity

Workshop Details

Workshop Plan: Part I

8:30–9:00 Introduction and core DID issues

9:00–9:45 Advanced DID and staggered adoption

9:45–10:30 Analysis 1: Advanced DID of COVID-19 vaccine mandates

Workshop Plan: Part II

10:40–11:15 Introduction to synthetic control

11:15–11:45 Analysis 3: SC of Ohio’s COVID-19 vaccine lottery

11:45–12:15 Advanced SC methods overview

12:15–12:30 Analysis 4: Advanced SC of multiple states’ COVID-19 vaccine lotteries

Workshop Goals

  • Understand, interpret, and critique the use of DID and SC in epidemiology

  • Gain familiarity with state-of-the-art methods related to DID and SC and identify resources for further exploration

  • Contextualize the assumptions needed for causal inference from quasi-experiments

  • Implement staggered adoption DID and SC analyses and diagnostics/inference in R

A Note on the Examples

I will focus here on infectious disease examples from published literature with available data. Some issues are specific to ID, while others are not, but they illustrate the points of how to approach these questions.

Title of Lopez Bernal et al. (2019)

Title for Kennedy-Shaffer (2024)

Title for Goodman-Bacon and Marcus (2020)

Title for Feng and Bilinski (2024)

Let’s Get To It!

All materials: https://github.com/leekshaffer/Epi-QEs/

QR code for above link

Epidemiologic Considerations

Key Assumptions

  • Parallel trends (in expectation of potential outcomes)

  • No spillover

  • No anticipation/clear time point for treatment

Methods to Improve Assumptions

  • Re-scale the outcome

  • Incorporate covariates

  • Include more or fewer units and/or time periods

ATT Estimand

Estimand Interpretation

DID estimates the Average Treatment Effect on the Treated (ATT).

This may not be generalizable to other units, including the untreated units in the study.

Internal vs. External Validity

  • Internal validity may be high if the assumptions are justified.

  • External validity may be low because of limited transportability of the ATT and limited information on effect heterogeneity.

Bias vs. Variance

  • Incorporating additional units/periods can reduce variance, but may also risk violating the assumptions

  • Generally conducted with limited, carefully-selected units: low bias but high variance

Examples

  • More distant vs. closer untreated units

  • Incorporating more untreated units

  • Incorporating more recent time periods

Summary: DID for Epidemiology

Advantages:

  • Simple to implement

  • Uses summary data

  • No need to model time trends or collect covariates

  • Straightforward interpretation

Disadvantages/Limitations:

  • Targets ATT not ATE

  • Need to justify key assumptions

  • Requires careful selection of controls

  • Limited inference with few units/periods

Questions